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Abstract 



1 Introduction 



We study the solution of ill-posed operator equations 

F{x)=y, 



(1) 



where F: X — > F is an operator between the topological spaces X and Y , and 
y £ F are given, noisy data, which are assumed to be close to some unknown, 
noise-free data j/^ = F{x^). If the operator F is not continuously invertible, 
then ID) may not have a solution and, if a solution exists, arbitrarily small 
perturbations of the data may lead to unacceptable results. 

If it is known that the given data satisfy an estimate ||?/^ — < (i^ one 
strategy for defining an approximate solution of ((T|) is to solve the constrained 
minimization problem 

7^(a;) ^ min subject to \\F{x)~y\\<l3. (2) 

Here, the regularization term TZ: X [0, +00] is intended to enforce certain 
regularity properties of the approximate solution and to stabilize the process of 
solving ^ . In [29l [45] , this strategy is called the residual method. It is closely 
related to Tikhonov regularization, which consists in minimizing the regulariza- 
tion functional 

r(x,y) \\F{x)-yf + anix) 

for some regularization parameter a > 0. 

While the theory of Tikhonov regularization has received much attention in 
the literature (see [Il[l[Ta[li[ll[2l[23[35l|4ll|4l|4i[48]),the same cannot be 
said about the residual method. Nevertheless, several results are available. The 
existence theory of ^ and also the question of convergence, which asks whether 
solutions of ^ converge to a solution of ([T]) as Hy — < f3 0, have been 
treated in a quite general setting in [55] (see also [HHIS]). Also, convergence 
rates have for instance been derived in [?j in a Hilbert space setting for a linear 
operator F and in O [7] for the reconstruction of sparse sequences. Still, no 
attempts have been made to carry over these results to more general spaces and 
functionals, as opposed to the recent developments in Tikhonov regularization 

(see [iniiMiiMiiMiiia). 

Even more, it seems that the problem of stability, that is, continuous depen- 
dence of the solution of ([2]) on the input data y and the presumed noise level (3, 
has hardly been considered at all. One reason is that, in contrast to Tikhonov 
regularization, stability simply does not hold for general non-linear operator 
equations. But even for the linear case, where we indeed prove stability, so far 
results are non-existent in the literature. 

The present paper intends to carry out the above indicated generalizations 
of the existent theory as far as possible. We assume that X and Y are mere 
topological spaces and consider the minimization of TZ{x) subject to the con- 
straint S[F{x),y) < j3. Here S is some distance like functional taking over the 
role of the norm in ((21) . In addition, we discuss the case where the operator F 
is not known exactly. This subsumes errors due to the modeling process as well 
as discretizations of the problem necessary for its numerical solution. 

We provide different criteria that ensure stability (Theorem 13.61 and Propo- 
sitions [3T0l|4]3]) and convergence (Propositions lS.Qi 14. 3p of the residual method. 
In particular, our conditions also include certain non- linear operators (see Ex- 
ample |421). Section [5] is concerned with the derivation of convergence rates. We 
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define a generalized Bregman distance tiiat allows us to state and prove rates 
on arbitrary topological spaces (see Theorem 15. Sp . In Section [6] we apply our 
general results to the case of sparse ^^'-regularization with p G (0, 2). We prove 
the well-posedness of the method and derive convergence rates with respect to 
the norm in a fairly general setting. In the case of convex regularization, that 
is, p > 1, we derive a convergence rate of order 0{S^^p). In the non-convex case 
< j3 < 1, we show that the rate 0{S) holds. 

2 Definitions and Mathematical Preliminaries 

Let X and Y be sets and -F: X ^ Y. Assume moreover that TZ: X ^ [0, +oo] 
and SiYxY^ [0, +00] is such that S{y,z) = if and only if y = z. We 
consider for given y (z Y and /3 > the constrained minimization problem 

TZ{x) min subject to S{F{x),y) < (i ■ (3) 

For the study of the properties of the solutions of ([3]), it is convenient to 
introduce the following notation. Let /? > 0, i > 0, y G F, and F : X — > F. We 
define the feasible set for the solution of ([3]) as 

HP,y,F) ■.= {xGX:S{F{x),y)<p] . 

In addition, we denote 

<i>K(/3, y, F, t) $(/?, y, F) n {x e X : 7^(x) < t] . 

The value of ([3]) is defined as 

v{P,y,F) inf{7^(x) : x e $(/3,y,F)} . 

The set of solutions of ([3]) is denoted by 

S(/3,y,F) {x G $(/3,y,F) : n{x) - v{P,y,F)] . 

Remark 2.1. An immediate consequence of the definition of y, F) is the 
identity 

I](/3,y,F) =ci>K(/3,y,F,i;(/3,y,F)) . 

The elements of S(0, y, F) satisfy F{x) — y and are referred to as TZ-minimizing 
solutions of the equation F{x) — y. ■ 

Lemma 2.2. The sets ^-n satisfy 

<i>K(/?,y,F,t) ca>K(/3 + 5,y,F,i + £) (4) 

for every £ > and 5 >Q, and 

<^nW,y,F,t)^ f] ^T^{P + S,y,F,t + e) . (5) 

<5,e>0 

In particular, 

S(/3, y , F) = fl $^ (P^ y,F,v{P,y,F)+e) . (6) 

e>0 
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Proof. The inclusion is a trivial consequence of the definition of ^u- For the 
proof of dn]) note that x e f]g $-r,(/3 + (5, y,F,t + e) if and only if S{F{x),y) < 
f3+S for all (5 > and TZ{x) < t+e for all e > 0. This, however, amounts to saying 
that S{F{x),y) < (3 and TZ{x) < t, which means that x G $-r,(/3, y, F, i). This 
proves one inclusion in ([S]), and the other inclusion is an obvious consequence 
of (21). Finally, equation © follows from Remark |2. II and ([5]). □ 

In the next section we study convergence and stability of the residual method, 
that is, the behavior of the set of solutions S(/3fc, Vk, F) for /3k ^ P and yk y. 
In [18l [26l |42] , where convergence and stability of Tikhonov regularization have 
been investigated, the results are of the form: every sequence {xk)keN with 
Xk & argmin{||F(a;) — yk\\^ + ak'R-{x)} has a subsequence {xkj)jeN converging 
to some element x € argmin{||_F(a;) — + aR-{x)}. We prove similar results 
for the residual method but with a different notation involving a type of con- 
vergence of sets (see [311 §29]). In addition, it is necessary to define a notion of 
convergence of {yk)keN in a way compatible with the distance measure S on Y. 

Definition 2.3. The sequence {yk)keN C Y converges S -uniformly to y G Y, if 

sup{|cS(2;,?;fc) - S{z,y)\ : zeY} ^0 . 

The sequence of mappings Fk: X Y converges locally S -uniformly to 
F: X -^Y, if 

sup{\S{Fkix),y) ~ SiF{x),y)\ ■.yeY,xeX, n{x) < i} ^ 

for t > 0. ■ 

Remark 2.4. If the distance measure S = d equals a metric on Y, then the 
5-uniform convergence of a sequence {yk)k&'N to y coincides with its convergence 
with respect to the metric. This result easily follows from the triangle inequality, 
as 

\d{z,yk) - d{z,y) \ < d{yk,y) 
for all z G Y. B 

Lemma 2.5. Assume that {yk)keN converges S-uniformly to y € Y and the 
mappings Fk'. X —^Y converge to F: X —^Y locally S-uniformly. Then there 
exists for every (3 > 0, t > 0, and s > some fco G N such that 

^n{P-e,y,F,t') c $k(/3, y^, i^^, <') 

C $-r(/3 + £, y,F,t') for every t' < t and k > ko . (7) 

Proof. Since yk y iS-uniformly and Fk ^ F locally iS-uniformly, there exists 
/co G N such that 

\S{Fk{x),yk)-S{Fk{x),y)\ < e/2 , 
\SiFkix),y)-SiFix),y)\<e/2, 
for all X € X with TZ{x) < t and k > ko. 
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Now let X e ^Ti{P - e, y, F, t). Then (O implies that 



\S{Fk{x),yk)-S{F{x),y)\ 

< \SiF,ix),yk)^SiFkix),y)\ + \SiFkix),y)~SiFix),y)\ <e, 

and thus 

S{Fk{x),yk) < S{F{x),y) + e < P , 

that is, X G yki Ft, t), which proves the first inclusion in ([7|). The second 

inclusion can be shown in a similar manner. □ 

Definition 2.6. Let r be a topology on the set X, and let Uk C X , k £ N, be 

a sequence of subsets of X. We define the upper limit of (t/fc)fceN as 



r-Lim supfc [/fc := Q fr-cl |J Uk 



fceN fc'>fc 

Here r-cl denotes the closure with respect to t. ■ 

Lemma 2.7. Let Uk C X, k € N, be a sequence of subsets of X . Then x € 
T-Limsupfc Uk, if and only if for every neighborhood N of x and every k € N 
there exists k' > k such that N (lUk' 7^ 0- 

Proof. This is a direct consequence of the definition of r-Lim supj. Uk ■ O 

Now assume that X satisfies the first axiom of countability, that is, every 
point X G X has a countable basis of neighborhoods, an assumption that is for 
instance satisfied for the weak topology on separable Banach spaces. Then one 
can characterize the upper limit of sets in terms of subsequences. 

Lemma 2.8. Assume that X satisfies the first axiom of countability. Then x G 
r-Lim sup^ Uk if and only if there exists a subsequence {Ukj)j^n and elements 
Xj € Ukj such that xj — >^ x. 

Proof See [HI §29.IV]. □ 

Definition 2.9. Let r be a topology on the set X, and let Uk C X , k Q N, be 
a sequence of subsets of X. An element x £ X is contained in the lower limit 
of the sequence {Uk)keN, in short, x e r-Lim inf^ Uk, if for every neighborhood 
N oi X there exists fceN such that N D Uk' for every k > k'. 

If the lower limit and the upper limit of the sequence {Uk)k&i coincide, we 
define r-Limj, Uk := r-Lim infj. Uk ~ r-Lim sup^. Uk. ■ 

Lemma 2.10. We have the characterization U = r-Lim^ Uk, if and only if 
every subsequence {Ukj)jet>i satisfies U ~ r-Lim sup^ Uk^- 

Proof See [SI §29. V]. □ 
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Lemma 2.11. Assume that the topology t on X is defined by a metric d. Then 
X G T-Linisupj, Uk, if and only if 

liminf dist(a;, Uk) = liminf \nf{d{x, u) : u £ Uk} = . 

k k 

Similarly, x e r-Liminffc Uk, if and only if 

limsup dist(a;, Uk) — limsup inf{d(a;, u) : w e Uk} = . 

k k 

Proof See [HI §29.I,§29.III]. □ 

The following lemma clarifies the relation between the stability and conver- 
gence results in [THl [ISl SI] and the results in the present paper. 

Lemma 2.12. Let Uk C X , k Cz N, be non-empty and assume that there exists 
a compact set K such that Uk CI K for all K . Then r-Limsupj, Uk is non-empty. 

If in addition, X satisfies the first axiom of countability, then every sequence 
Xk G Uk has a subsequence converging to some x G T-Lim sup^, Uk ■ 

Proof. By assumption, the sets Sk := ''"-cllj/£'>fe Uk form a decreasing family of 
non-empty, compact sets. Thus also their intersection HfeeN ~ T-Lim supj, Uk 
is non-empty (see [SO] Thm. 5.1]). 

Now assume that X satisfies the first axiom of countability. Then in par- 
ticular every compact set is sequentially compact (see [501 Thm. 5.5]). Let now 
Xk S Uk, fc G N. Then the sequence {xk)k&i has a subsequence {xkj)j&i con- 
verging to some X (z K. From Lemma 12.81 we obtain that x G r-Lim sup^, ?7fc j 
which shows the assertion. □ 



3 Well-posedness 

In the following we investigate the existence of minimizers, and the stability 
and the convergence of the residual method. Throughout the whole section, we 
assume that {X,t) is a topological space, Y a set, F: X ^ Y some operator, 
2/ G y, and /3 > 0. 

Theorem 3.1 (Existence). Assume that ^n{[3,y,F,t) is t- compact for every 
t and non-empty for some t. Then Problem ^ has a solution. 

Proof. Remark 12.11 and (O show that 

E(/?,y,F)= {~]^n{P,V.F,v{p,y,F)+e) 

is the intersection of a decreasing family of non-empty r-compact sets and thus 
non-empty (see [501 Thm. 5.1]). □ 



6 



Remark 3.2. Recall that a mapping T : X ^ [0, +00] is called lower semi- 
continuous, if its lower level sets {x S X : T[x) < are closed for every t > 0. 
Moreover, it is coercive, if the lower level sets are pre-compact. Thus, T is lower 
semi-continuous and coercive, if and only if its lower level sets are compact. 
Since the intersection of a closed set and a compact set is itself compact, the 
sets $7j(/3,y,F, are r-compact, if both mappings TZ and x i— > S[F{x),y) are 
lower semi-continuous and at least one of them (or their sum) is coercive. 

The lower semi-continuity of a; 1— > S(^F{x), j/) certainly holds if F is contin- 
uous and S lower semi-continuous with respect to the first component. It is, 
however, also possible to obtain lower semi-continuity, if F is not continuous 
but the functional S satisfies a stronger condition. Assume therefore that the 
mapping z 1-^ S{z,y) is lower semi-continuous and coercive, and F: X ^ Y has 
a closed graph. Then the set {z £Y : S{z, y) < /?} is compact for every /3 > 0. 
Because F has a closed graph, the pre-image under F of every compact set is 
closed (see [28l Thm. 4]). This shows that {x £ X : S{F{x),y) < /?} is closed 
for every /3, that is, the composition x 1-* S{F{x),y) is lower semi-continuous. a 

Stability is concerned with the continuous dependence of the solutions of ([3]) 
of the input data, that is, the element y, the parameter /3, and, possibly, the 
operator F. Given sequences Pk ^ P, Vk ^ y, and Fk F, we ask whether 
the sequence of sets T,{f3k,yk, Fk) converges to S(/3, y, F). As already indicated 
in Section [21 we will make use of the upper convergence of sets introduced in 
Definition 12.61 The topology, however, with respect to which the results are 
formulated, is finer than r. 

Definition 3.3. The topology r-n on X is generated by all sets of the form 
U n {x e X : s < TZ{x) < t} with U € t and s < t e R. A sequence 
(xfc)fcgN C X converges to x with respect to t-jz, if and only if Xk — >r x and 

TZ{xk) ^ n{x). ■ 

Below we provide conditions that guarantee upper semi-continuity of the set 
of solutions with respect to t-jz in the sense that 7^ TT^-Limsup^. l^{(3k,yk, -Ffc) C 
T,{P,y, F). That is, the minimizing sets for Pk, yk, and Fj- converge to the 
minimizing set for /3, y, and F. If T,[p,y,F) consists of a single element x\ 
then this already implies that every sequence of approximate solutions converges 
to x^. Before proving these results, we require an additional lemma stating that 
the value of the minimization problem ([3]) behaves well as the regularization 
parameter /3 decreases. 

Lemma 3.4. Assume that $-^,(7, y, F, t) is r-compact for every 7 and every t. 
Then the value v of ^ satisfies 

viP,y,F)^supviP + e,y,F) . (9) 

e>0 

Proof Since ^niP, V, F, t) c $7^(/3 -I- e, y, F, t), it follows that 

viP,y,F)>v{p + e,y,F) 
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for every £ > 0, and therefore v{p, y, F) > sup^^g v{f3 + e, y, F). 

In order to show the converse inequahty, let ^ > 0. Then the definition of 
v{P, y, F) implies that ^n{l3, y, F, v{l3, y, F) - 6) ^ 0. Since (cf. Lemma 

9 = 'pTz{P,y,F,v{p,y,F)-S) = f] 'S>n{P + e,y, F,viP,y, F) - S) (10) 

and the right hand side of pI7|) is a decreasing family of compact sets, it follows 
that already ^tz[P + e, y, F, v{P, y,F) ~ 5^ —% for some e > 0, and thus 

v{l3 + e,y,F)>v{l3,y,F)-5 . 

Since 5 was arbitrary, this shows the assertion. □ 

For the main stability results we make the following assumption: 

Assumption 3.5. Let /3 > 0, {Pk)ken be a sequence of non-negative numbers, 
y eY, {yk)ken C Y, and F, Fk: X ^ Y, k e N. The sets <i>Tz{l,w, Fk,t) and 
^Tzili F, t) are compact for all 7, w, t, and k and non-empty some t. 

Theorem 3.6 (Stability). Let Assumption \3.5\ hold. Assume that {yk)kGfi 
converges S -uniformly to y Cz Y , the mappings Fk'. X Y converge locally 
S -uniformly to F : X Y , and (3k (3. If 

limsupw(/3fc,yfc,Ffc) < v{f3,y,F) < 00, (11) 

k 

then 

^ TTz-UmsnVkn^k.yk^Fk) C S(/3,2/,F) . (12) 
If the set T,(P,y,F) consists of a single element xp, then 

{xp} = r-Tj-Limfe T,{Pk, yk, Fk) ■ (13) 

Proof. Define the set T :— r-Linisup^. I](/3/c, y^, -Ffc)- Because the topology t-ji 
is finer than r, it follows that TT^-Limsupj. E(/3fc, y^, i^^) C T. We proceed by 
showing that T C Y,{(3,y,F) and T C r-Tj-Limsup;. S](/3fc, y/j, F^), which 
then gives the assertion (|12p . 

In order to simplify the notation, we define 

$fc(t) :=$7j(/3fc,2/fc,ft,t), m ■.^<^n{P,y,F,t), 
Vk ■.= v{f]k,yk,Fk) , v:=v{P,y,F) , 

^k-^nfik,yk.Fk), E:=S(/3,y,F) . 

The inequality (fTTj) implies that for every e > there exists some fco S N 
such that Wfe < w + e for all k > k^. Since (3k ~^ (3, we may additionally assume 
that (3k < P + e. Applying Lemma [^31 we see that, after possibly enlarging fco: 

$fc(ufe) C <i>n{(3 + e,yk,Fk,Vk) 

C $7j(/3 + 2e, y, F, vk) C <i>n{P + 2e, y,F,v + e) (14) 
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for all k > kci. Thus, 

T = r-Limsupfc = Q (r-cl (J Sfe'^ 

fceN k'>k 

= n (^-'^l U C $7j(/3 + 2£,2/,i^,,; + e) . (15) 

k>ko k'>k 

The sets r-clljj,,>^ E^,/ are closed and non-empty and, by assumption, the set 
^7i(/3+2e, y, F, v+e) is compact. Thus T is the intersection of a decreasing fam- 
ily of non-empty compact sets and therefore non-empty. Moreover, because (jlSp 
holds for every e > 0, we have 

0^Tc f]^TziP + 2e,y,F,v + e)^^v) = J: . (16) 

Next we show the inclusion T C TT^-Limsup;. S^. To that end, we first prove 
that 

V = limufc . (17) 

k 

Recall that Theorem 13.11 implies that ^k{vk) = Sfe is non-empty. Therefore, 
implies that also $-r,(/3 + 2e,y, F,Vk) is non-empty, which in turn shows 
that Vk > v{(3 + 2e, y, F) for all k large enough. Consequently, 

\i-m\nivk>v{P + 2e,y,F) (18) 

k 

for all £ > 0. From Lemma 13.41 we obtain that v — supg>o''^(/5 + 2e, y,F). 
Together with (HHD and ^ this shows (fT7)) . 

Let now x ^ N, let iV be a neighborhood of x with respect to r, let 5 > 
and fco e N. Since T C E (see (HH)), it follows that 7e(a;) = v. Thus it follows 
from (fT7|) that there exists fci > fco such that 

|^;fc-7^(x)| < S 

for all k> ki. In particular, 

Efc C {i e X : 7^(x) - J < 7^(i) < 7^(a;) -f 5} (19) 

for all k>ki. Lemma 12.71 implies that there exists k2 > ki such that 

iV n Efe, ^ . (20) 

Now recall that the sets N n {x e X : n{x) - S < 7^(^) < n{x) + 5} form 
a basis of neighborhoods of x for the topology t-ji. Therefore (fTO|) . ([^0]) . and 
the characterization of the upper limit of sets given in Lemma 12.71 imply that 
X G TTj-Limsup;, E^. Thus the inclusion (I12p follows. 

If the set E(/3, y, F) consists of a single element xp, then the first part of the 
assertion implies that for every subsequence (fcj)j(=N we have 

r-R,-Lim sup^- E(/3fc^ , yk^ , Fk- ) = {x/s} . 

Thus the assertion follows from Lemma [2. 101 □ 
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The crucial assumption in Theorem 13.61 is the inequahty ([TT|) . Indeed, one 
can easily construct examples, where this condition fails and the solution of 
Problem ^ is unstable (see Example 13 . 71 below) . What happens in the example 
is that TK-Limsup;. T,{(3k,yk, F) consists of local minima of TZ on $(/?, y, F) that 
fail to be global minima of TZ restricted to $(/3, y, F). 



Figure 1: The nonlinear function F from Example 13.71 The set 'S{/3,y,F) 
consists of an interval and the isolated point {0}. 

Example 3.7. Consider the function F: M ^ R, F{x) — — x'^, and the 
regularization functional TZ{x) ~ x^. Let y > and choose (3 ~ y. Then 

argmin{7?.(a;) : \F{x) - y| < /?} = argminjx^ ■ - x'^ - y \ < y} ^ ■ (21) 

Now let yk > y. Then 

argmin{7?.(2;) : \F{x) - yk\ < P} ^ argminja;^ : \x^ - x'^ - yk\ < y} = Xk , 

where Xk is the unique solution of the equation F{x) = yk — y- Thus, if the 
sequence {yk)k(EN converges to y from above, we have Xk > 1 for all k and 
liTakXk — 1. According to (|2ip . however, the solution of the limit problem 
equals zero. ■ 

In the above example the solution is unstable, because the feasible set 
$(/3, y, F) for y contains elements that cannot be reached by the sets ^{f3k,yk, F). 
As a consequence, the limit of the sets S(/3fe, yk, F) consists of local minima of 
the limit problem instead of global ones. The next result shows that by only 
slightly increasing the parameters (3k, the feasible sets ^{(3k,yk, F) becomes 
sufficiently large as to contain the solution set 'E,{(3,y,F). 

Proposition 3.8 (Approximate Stability). Let Assuinvtion \3.5\ hold. As- 
sume that {yk)keN converges S-uniformly to y Cz Y , the mappings Fk'. X 
converge locally S-uniformly to F: X Y, and [3k ^ (3. Then there exists a 
sequence Sk such that 








1 



^ r7^-Limsupfe Y,{(3k + Sk, yk,Fk) C y, F) . 
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Proof. Define 

Ek inf {e > : <Pn (/?, V, F, y, F)) C {h + e, 2/fc, i^fc, t'C/?, y, F)) ] . 

Lemma 12.51 and the assumption that /3k ^ (3 imply that Sk ^ 0- Since by 
assumption 

^ S(/3, y, F) = $^ (/3, F, z;(/3, F)) C <i>K(/3fc + e^, ^fe, F^, y, F)) , 

we obtain that v{(3k + £k,yk, Fk) < u(/3, y, F). Thus the assertion foUows from 
Theorem ESI □ 



Proposition 3.9 (Convergence). Let Assumvtion U75\ hold. Assume that the 
sequence {yk)keTi converges S-uniformly to y £ Y and S{y,yk) < Pk ~* 0. 
Assume moreover that there exists x Cz X with TZ{x) < oo and F{x) — y. Then 

\imsnpv{pk,yk,F)<v{0,y,F) . (22) 

k 

In particular, 

^ r7z-LimsupfcE(/3fc,yfe,F) C S(0,y,F) . (23) 
// the set S(0, y, F) consists of a single element x\ then 

{xt} = r7z-LimfcI](/3fc,yfc,F) . (24) 

Proof. By assumption S{y,yk) < Pk, which imphcs that v{Pk,yk, F) < TZ{x') 
for all x' e $(0,j/,F). This proves (EH). Now ^ and ^ follow from Theo- 
rem [Ml □ 



Proposition 3.10 (Stability). Let Assumvtion \3.5\ hold. Assume that the se- 
quence {yk)keti converges S-uniformly to y G Y, the mappings Fk'. X Y 
converge locally S-uniformly to F : X ^ Y , and f3k P > 0- Assume moreover 
that 

^n{P. y, F,t)(i{~] (r-cl U ^niP -e,y,F,t + 6)) (25) 

<5>0 e>0 

for every t> Q. Then 

\irasupvif3k,yk,Fk) <v{l3,y,F) . (26) 

k 

In particular, 

^ r7z-LimsupfeS(/3fc,yfc,Ffe) C S(/3,2/,F) . (27) 
// the set S(/?, y, F) consists of a single element xp, then 
{xp} = TTj-Limfc T.(l3k, yk, Fk) . 
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Proof. The convergence of (/3fe)feeN to /3 and Lemma [2.51 imply that for every 
e > and t G R there exists fcg € N such that 

^n{P-e,y,F,t) c <i>n{Pk,yk, Fk,t) 

for all k > ko. Consequently, 

limsup v{Pk,yk,Fk) = limsup m{{t : ^n{Pk,yk, Fk,t) ^ 0} 

k k 

< inf M{t ■.<i>n{P-e,y,F,t)y^(d}. 

From P5|) we obtain that 

inf inf {i : ^niP - £, 2/, F, t) ^ 9} < inf : 2/, i) 0} = 2/, F) . 

This shows dill)- Now ^ follows from Theorem [3H □ 

4 Linear Spaces 

Now we assume that X and Y are topological vector spaces. Then their linear 
structure allows us to introduce more tangible conditions implying stability of 
the residual method. 

Assumption 4.1. Assume that the following hold: 

1. The sets X and Y are topological vector spaces. 

2. For all xq, xi d X with S(^F{xo), y) , S(^F{xi),y) < oo, and all < X < 1 
we have 

S{F{Xxo + il- X)xi),y) <max{S{F{xo),y),S{F{xi),y)} . 

Moreover, the inequality is strict for all < X < 1 whenever S(^F(xo), y) 7^ 
S{F{x^),y). 

3. For every /3 > there exists x G X with < f3 and TZ{x) < 00. 

4- The domain domT?. — \^x E X : TZ{x) < +00} 0/ 7?. is convex and for 
every xq, xi E domT?., the restriction of TZ to 

L ^ {Xxo + (1 - A)a;i : < A < l} 

is continuous. 

We now show that the Assumption 14.11 implies the main condition, the in- 
clusion (PS)) . of the stability result Proposition 13. 101 

Lemma 4.2. Assume that Assumption \4. 1\ holds. Then (j25p is satisfied. 
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Proof. Let xq G ^n{P, y, F, t) for some /3 > 0. We have to show that for every 
neighborhood N C X of xq and every S > there exist e > and x' G N such 
that x' e $tc(/3 ~e,y,F,t + 6). 

Item 13] in Assumption 14.11 impUes the existence of some xi ^ X satisfying 
S{F(xi),y) < P and 7^(xl) < oo. Since S{F{xi),y) < /3 and S{F{xQ),y) < /3, 
we obtain from Item[2]that S{F{x), y) < (i for every x £ L -.^ {Xxq + (1 — A)a;i : 
< A < l}. Since xq, xi £ domT?., it foUows from Item [5] that TZ is continuous 
on L. Consequently Um^^i 7?,(Aa;o + (l — A)a;i) —TZ{xo) < t. In particular, there 
exists Ao < 1 such that TZ{Xxo + (1 — A)a;i) < i + (5 for all 1 > A > Aq. Since X 
is a topological vector space (Item[T]), it follows that x' := Xxq + (1 ^ A).ti G N 
for some 1 > A > Aq. This shows the assertion with e := /? — S(^F{x'), y) > O.D 

Lemma 14.21 allows us to apply the stability result Proposition I3.10i which 
shows that Assumption 14.11 implies the continuous dependence of the solutions 
of ((HI) on the data y and the regularization parameter (3. 

Proposition 4.3 (Stability &: Convergence). Let Assuinption \jA\ hold and 
assume that ^■ji{'-/,w, F,t) is compact for every 7 > 0, t G K, and w £ Y . 
Assume moreover that {yk)keN converges S-uniformly to y £ Y , and j3k — > (3- 
If (3 — Q, assume in addition that S{y,yk) < Pk- Then 

0^TK-LimsupfcE(/3fc,j/fe,F) C ^iP,y,F) . 

If the set y, F) consists of a single element xp, then 

{xfj} = TTz-Umk^{l3k,yk, F) . 

Proof. If /3 = 0, the assertion follows from Proposition 13.91 In the case /3 > 0, 
Lemma [4.21 implies that (l25t holds. Thus, the assertion follows from Proposi- 
tion [3ll0l □ 

Proposition 4.4 (Stability). Let Assumption \jA\ hold. Assume that {yk)k&i 
converges S-uniformly to y £ Y , the mappings Fk'. X Y converge locally 
S-uniformly to F: X ^ Y (see Definition \2.3\) . and (3k (3 > 0. Assume that 
the sets {x £ X : TZ{x) < t} , ^nilTW, Fk,t) and ^iz{'y,w,F,t) are compact for 
every 7 > 0, t G K, and w £ Y . Then 

^ TK-Limsupfc yk, Fk) C y, F) . 

If the set Ti{(3, y, F) consists of a single element xp, then 

{xp} = T-ji-Limk T.{(3k, yk, Fk) ■ 

Proof. Again, Lemma 14.21 shows that ([25]) holds. Thus the assertion follows 
from Proposition l3.10l □ 

Item [2] in Assumption 14.11 is concerned with the interplay of the functional 
F and the distance measure S. The next two examples consider two situations. 
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where this part of the assumption holds. Example 14. 51 considers linear operators 
F and convex distance measures S. Example |4]6] introduces a class of non-linear 
operators on Hilbert spaces, where Item [5] is satisfied if the distance measure 
equals the squared Hilbert space norm. 

Example 4.5. Assume that F: X ^ Y is linear and S is convex in its first 
component. Then Item [5] in Assumption 14.11 is satisfied. Indeed, in this case 

S{F{Xxo + (1 - X)xi),y) = S{XF{xo) + (1 - \)F{xi), y) 
< \S{Fixo),y) + (1 - \)S{Fixi),y) < max{5(F(a:o), y) , y) } . 

If moreover, S[F{xo), y) ^ S(F{xi), y) and < A < 1, then the last inequality 
is strict. B 

Example 4.6. Assume that F is a Hilbert space, S{y,z) = \\y — z\\, and 
F: X ^ Y is two times Gateaux differentiable. Then Item[2]in Assumption 14. II 
is equivalent to the assumption that for all xq, xi € X the mapping 

t^T{t;xo,Xi) := \\F{xo + txi) - yf 

has no local maxima. This condition holds, if dfT{0;xo,xi) > whenever 
dtT{0; xq, xi) — 0. The computation of the derivatives of T( - ;xq,xi) at zero 
yields that 

dtT{0;xo,xi) = 2{F'{xo)ixi),F{xo)) 

and 

dfT{0; xo, xi) = 2{F"{xo)ixi:xi),F{xo)) + 2\\F'{xo)xi f . 

Consequently, Item [2] in Assumption 14. ll is satisfied if, for every xq, xi d X 
with xi ^ 0, the equality (^F' {xq)(xi), F{xq)'^ = implies that 

{F"{xo){xuXi),F{xo)) + ||F'(a;o)(xi)f > . , 

Example 4.7. Let p > 1 and X — L''{il,fj.) for some cr-finite measure space 
(17, /i). Assume that F is a Banach space and F : X ^ Y is a bounded linear 
operator with dense range. Let Ti{x) = ||a;|l^ and S{w,y) — \\w — y\\. We thus 
consider the minimization problem 

^ min subject to \\Fx — y\\ < (3 . 

We now show that in this situation the assumptions of Proposition 14.31 are 
satisfied. To that end, let r be the weak topology on As L^(ri, /i) is 

reflexive, the level sets {x £ X : TZ{x) < are weakly compact. Moreover, 
the mapping x '—^ \\Fx — y\\ is weakly lower semi-continuous. Thus all the 
sets $7^(7, w, F, i) are weakly compact. Example 14.51 shows that Item [2] in 
Assumption l4. ll holds. Item [3] follows from the density of the range of F. Finally, 
Item m holds, because TZ is norm continuous and convex. 
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Now assume that i/k y and 13^ ^ (3. If /3 = assume in addition that 
lli/fc ~ y|| < The strict convexity of TZ and convexity of the mappings x i— *■ 
\\Fx — yk\\ imply that each set T,{f3kTyk, F) consists of a single element Xk- 
Similarly, T,{i3,y,F) consists of a single element a;^. From Proposition 14.31 we 
now obtain that {xk)kGN weakly converges to a;^ and ||a;fe||^ l|a;^||p- Thus, in 
fact, the sequence {xk)kGN strongly converges to x^ (see [SH Cor. 5.2.19]). 

Let P > and assume that : X — > F is a sequence of bounded linear 
operators converging to F with respect to the strong topology on L{X, Y), that 
is, sup{||Ffca; — Fx\\ : \\x\\ < 1} ^ 0. Let again ^ /? and yk y, and denote 
by Xk the single element in S(/3fe, j/j., i^j,). Applying Proposition 14. 4[ we again 
obtain that a:^ — > . ■ 

Remark 4.8. Example 14.71 relies heavily on the assumption that p > 1, which 
implies that the space is reflexive. In the case X = L^(J7,/i), the level 

sets {x & X : \\x\\i < fail to be weakly compact, and thus even the existence 
of a solution of Problem ([3]) need not hold. ■ 

Remark 4.9. The assertions of Example 14.71 concerning stability and conver- 
gence with respect to the norm topology remain valid, if X is any uniformly 
convex Banach space and TZ the norm on X to some power p > 1. Also in 
this case, weak convergence and convergence of norms imply the strong conver- 
gence of a sequence [Ml Thm. 5.2.18]. More generally, this property is called 
the Radon-Riesz property [HU p. 453]. Spaces satisfying this property are also 
called Efimov-Stechkin spaces in [53]. ■ 



5 Convergence Rates 

In this section we derive quantitative estimates (convergence rates) for the dif- 
ference between regularized solutions Xfj € S(/3, y, F) and the exact solution of 
the equation F{x^) = y^ . 

For Tikhonov regularization, convergence rates have been derived in [H \26\ 
I39j in terms of the Bregman distance. However, its classical definition, 

D^{x, x^) = n{x) ~ (7^(a;t) + {^,x~ x^)x*,x) , 

where ^ G &R.{x^), requires the space X to be linear and the functional TZ to be 
convex, as the subdifferential dTZ{x^) is only defined for convex functionals. In 
the sequel we will extend the notion of Bregman distances to work for arbitrary 
functionals TZ on arbitrary sets X . 

Definition 5.1 (Generalized Bregman Distance). Let??.: X MU{+oo} 
and let 

x^ e dom(7^) -.^ {xe X : TZ{x) < oo} 
be an element in its domain. 
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A functional T: X ^ R U {+00} is called a Bregman tangent for TZ at if 
T{x^) 0, dom(7^) C dom(r), and the mapping 

.^7^.(.,.t)^^/^(-)-(^(-^) + ^(-))> if.Edom(7e), 

^ ^ [+00, if a; e X \ dom(7^) , ^ ' 

is non-negative. The mapping Dt[ ■ tX^) is called the Bregman distance corre- 
sponding to T. B 

Remark 5.2. Let X be a Banach space, let TZ be convex, and let ^: X ^ R 
be a bounded linear functional. Then the mapping 

re(x) :-(e,x-xt) (29) 

a is a Bregman tangent for 7?, at , if and only if ^ S dTZ{x'^). Thus, the standard 
Bregman distance I?^ = Dq-^ is indeed a special case of our generalized notion. > 

Convergence rates in terms of the Bregman distance will be obtained 
under the following assumption: 

Assumption 5.3. 

1. There exists a monotonically increasing function 6 : [0,oo) [0, cxd) such 
that 

S{W1,W2) < 9{S{W1,W3) +S{W2,W3)) (30) 

for all wi, W2, W3 ^ Y . 

2. There exists an element x^ G dom(7?.), a Bregman tangent T for TZ at x\ 
and constants 71 £ [0, 1) and 72 > such that 

-T{x) < ^iDrix, x^) + j2S{F{x), F(xt)) (31) 

for every x G $k(6'(2/5), F(a;t), F,7^(xt)) . 

Remark 5.4. In a Banach space setting (see Subsection 15.11 below) . the source 
inequality ((3T|) has already been used in [26l |42] to derive convergence rates for 
Tikhonov regularization with convex functionals. ■ 

Theorem 5.5 (Convergence Rates). Let Assumvtion [5.S\ hold. Then 

sup{Dr{xf3,x^) : xp £j:iP,y,F)} < -^9{p + S{F{x^),y)) . (32) 

i — 71 

for ally eY with S{F{x'<), y) < [3. 

Proof. Let xp G T,{/3,y, F). This, together with ([50]) and the assumption that 
S{F{x^),y) < f3, implies that 

S{F{xp),F{x^)) < e{S{F{xp),y)+S{F{x^),y)) < 0{2f3) . 
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Thus we can apply ([31]) and the definition of the Bregman distance, to deduce 
that 

- T{x0) < jiDT{xf3,x^) + -f2S{F{xf3),F{x^)) 

= li{n{xp) - 7^(xt) - T{xp)) + ^2S{F{xp),F{x^)) . 

Together with the assumption 71 G [0, 1) this shows the inequaUty 

~r{xp) < -J^{Tl{xp) -Tl{x^)) + -^S{F{xp),F{x^)) . (33) 
-1- — 71 -1- ~ 71 

Since S{F{x'^),y) < f3, it foUows that TZ{xp) - TZ{x^) < 0. Therefore §^ 
imphes that 

Dt{xp,x^) < S{Fixf3),F{x^)) . (34) 

i — 71 

Consequently we obtain from pO|) and the estimate 5(F(a;^),y) < (3 the re- 
quired inequality 

Dt{x0, x^) < e{p + S{F{x^),y)) . □ 

1 - 71 

Remark 5.6. Typically, convergence rates are formulated in a setting which 
slightly differs from the one of Theorem 15.51 (see [H [TSl UHl SI])- There one 
assumes the existence of an 7?.-minimizing solution x^ d X of the equation 
F{x^) = y\ for some exact data y'^ G Y. Instead of y\ only noisy data y S V 
and the error bound S{y^,y) < (3 are given. For this setting, ([32|l implies the 
rate 

Dt{xp,x'^) < -^^B{2(3) =0{9{2(3)) as /3 ^ , 
1 — 71 

where xp € S(/3, y, F) denotes any regularized solution. ■ 



5.1 Convergence rates in Banach spaces 

In the following, assume that X and Y are Banach spaces with norms 
and II'IIy, and set S{y,z) :~ \\y — z\\y. Let 7?. be a convex and lower semi- 
continuous functional on X, and let :— Dt^ with ^ € dTZ{x'') denote the 
classical Bregman distance (see Remark l5.2p . 

Given data y satisfying ||_F'(x^) — y\\ < /3, Theorem 15.51 implies the conver- 
gence rate D^{xi3,x'^) = 0{(3), where £ T,{P,y,F) is a regularized solution 
and satisfies the source inequality 

-{^,x~x^)<^,D^{x,x^)+j2S{F{x),F{x^)). (35) 

Equation ([55]) has already been used in [53] to derive convergence rates for 
Tikhonov regularization. 

In the special case where X is a Hilbert space and Tl{x) = \\x\W/2 we have 
D^{x,x'') = \\x — x^|j^/2, which implies the rate 0{y/P) with respect to the 
norm. In Proposition 15.81 below we show that this rate holds on any 2-convex 
space. For r-convex Banach spaces with r > 2, we derive the rate 0{S^^^). 
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Remark 5.7. The book [321 PP- 70ff| clarifies the relation between ([35]) and 
the source conditions used to derive convergence rates for convex functional 
on Banach spaces (see jH [3^). In particular, it is shown that, if F and TZ 
are Gateaux differentiable at and there exist 7 > and uj ^ Y* such that 
7||w||i'. < 1 and 

e = F'(xt)*a;ea7^(x^), (36) 
||F(a;) - F{x^) ~ F'{x''){x - < -fD^{x, x^) (37) 

for every x e X, then holds on X. Here F'{x'')* : Y* ^ X* is the adjoint 
of F'(a;^). Conversely, if C G dTZ{x'^) satisfies (l35t . then (|36| holds for every 

In the particular case that F : X ^ Y linear and bounded, the inequal- 
ity ([37]) is trivially satisfied with 7 = 0. Thus, ([35)1 is equivalent to the source- 
wise representability of the subgradient, 

e e dnix^) n ran(F*) . (38) 

Here ran(i^*) = {F'{x'^)*uj : uj e X*] denotes the range of F'(a;t)*. , 

Recall that the Banach space X is called r-convex (or is said to have modulus 
of convexity of power type r) , if there exists a constant C > such that 

inf{l - II {x + y)/2|| : ||x|| = ||y|| = 1, ||x - y\\ > e} > Ce^ 

for all £ € [0,2]. Notice that all Hilbert spaces are 2-convex and that there is 
no Banach space (of dimension > 2) that is r-convex for some r < 2 (see |32l 

pp. 63flq). 

Proposition 5.8 (Convergence rates in the norm). Let X he an r-convex 
Banach space with r > 2 and let TZ{x) :~ ||x||5f/r. Assume that there exists 
x'^ e X , a subgradient ^ £ dTZ{x''), and constants 71 G [0, 1), 72 > such that 
([35]) holds for every x € $^(2/3, i^(a;^), F, 7^(xt)) . 
Then there exists a constant c > such that 

sup{||x^-a;t|| :.T;3eE(/3,y,F)} <c(/3+||F(xt)-y||)'/'^ (39) 

for all (3 > and y eY with \\F{x^) - y\\ < (3. 

Proof. Let Jr : X ^ 2^ denote the duality mapping with respect to the 
weight function s s^^^. In [321 Equation (2.17)'] it is shown that there exists 
a constant K > such that 

ll^;^ + 4x> W^^Wx + r{jr{x^), z)x,X' + i^ll^ll^f (40) 

for all x'l', z € X and ir{x'^) G Jr{x'^)- By Asplund's theorem [TI] Chap. 1, 
Thm. 4.4], the subgradient of TZ{x) = ||a;||''/r equals the duality mapping Jr. 
Therefore, by taking z — x — x'^ and jr{x^) — inequality ([40]) implies 

D^{x,x'')>K/r\\x~x^\'x for all a;^ x G X and C € 97^(xt) . (41) 

Consequently, ([35)1 follows from Theorem 15.51 □ 
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Exact values for the constant K in (|¥T|) (and thus for the constant c in ([5^ ) 
can be derived from [49] . Bregman distances satisfying (|4T|) are called r-coercive 
in [55] . This r-coercivity has already been applied in [3] for the minimization of 
Tikhonov functionals in Banach spaces. 

Example 5.9. The spaces X = U'{Vl,^) for p € (1,2) and some cr-finite mea- 
sure space (f^, ^) are examples of 2-convex Banach spaces (see [311 P- 81, Re- 
marks following Theorem l.f.l.]). Consequently we obtain for these spaces the 
convergence rate O{yf0) . The spaces X — LP{Q, /i) for p> 2 are only p-convex, 
leading to the rate ©(/J^/^) in those spaces. ■ 

6 An Application: Sparse Regularization 

Let A be an at most countable index set, define 

A 

and assume that F: X := £'^{A) ^ F is a bounded linear operator with dense 
range in the Hilbert space Y. We consider for p £ (0, 2) the minimization 
problem 

7^p(a;) := ||a;||^,(^) ^la^AT^min subject to \\Fx - y\\l^ < /3 . {A2) 

AgA 

For p > 1, the subdifferential dTZp{x^) is at most single valued and is identified 
with its single element. 

Remark 6.1 (Compressed Sensing). In a finite dimensional setting with 
p = 1, the minimization problem (|42p has received a lot of attention during the 
last years under the name of compressed sensing (see [5l l6l [8l [Ml [TSl [TBI [211 147] V 
Under some assumptions, the solution of (|42l) with y = Fx^ and /3 = has been 
shown to recover x^ exactly provided the set {X E A : x\^ ^ 0^ has sufficiently 
small cardinality (that is named, it is sufficiently sparse) . Results for p < 1 can 
be found in [i [131 [201 HO]. ■ 

In this section we prove well-posedness of (|42l) and derive convergence rates 
in a possibly infinite dimensional setting. This inverse problems point of view 
has so far only been considered for Tikhonov regularization 

TZp{x) + a\\Fx — yWy rnin 

(see [10l[l2l[22l[23l[33l[37l[5l]). 

In the following r denotes the weak topology on f^(A), and Tp := t-ji denotes 
the topology as in Definition 13.31 Then a sequence {xk)kGfii C ^^(A) converges 
to X S ^^(A) with respect to Tp if and only if x^ x and TZp(xk) — > TZ{x). As 
shown in [23, Lemma 2] this already implies TZp{xk ^ x) 0. In particular, the 
topology Tp is stronger than the topology induced by ||-||^2(a'). 
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6.1 Convex P Regular izat ion 

We first assume that p £ [1,2), in wliicli case the functional 72.p is convex. 

Proposition 6.2 (Well-Posedness). Letl<p<2 and let F-A'^{K) he 
a hounded linear operator with dense range. Then the following hold: 

1. Existence: For every /3 > and y Cz Y , the set of regularized solutions 
Yi{l3,y,F) is non-empty. 

2. Stabihty: Let {f3k) o,nd [yk) be sequences with — > /3 > and yk ^ y & 
Y. Then ^ Tp-Umsnp^ N{Pk,yk, F) C N{f3,y,F). 

3. Convergence: Let \\yk — y||y < /3fc — > and assume that the equation 
Fx = y has a solution in P'{h). Then ^ Tp-Limsupj, ?/fe, -F) C 
E(0,2/, -F). // the equation Fx = y has a unique TZp -minimizing solution 
x^ , then Tp-Lim^T,{Pk,yk, F) = {x^. 

Remark 6.3. In the case p > I, the functional TZp is strictly convex, and 
therefore the 7?.p-minimizing solution x^ of Fx = y is unique. Consequently 
the equality Tp-Lim^ 'E,{(3k,yk, F) = {x^} holds for every y in the range of the 
operator F. ■ 

Proof f of Proposition [6.2\) . In order to prove the existence of minimizers, we 
apply Theorem 13.11 bv showing that ^n{(3,y,F,t) is compact with respect to 
the weak topology on ^^(A) for every t > and is nonempty for some t. Because 
F has dense range, the set 

$k(/3, y, F, t)^{xe e{A) : TZpix) < t, \\F[x) ~ yfy < P] 

is non-empty for t large enough. It remains to show that the sets <I'k(/3, y, F, t) 
are weakly compact on ^■^(A) for every positive t. 

The functional TZp{x) = SagaI-^-'^I^ weakly lower semi-continuous (on 
^^(A)) as the sum of non-negative and weakly continuous functionals (see [T7]V 
Moreover, the mapping F is weakly continuous, and therefore x i-^ \\Fx — 
yWy is weakly lower semi-continuous, too. The estimate TZp{x) > \\x\\^2(^j^-^ 
(see [231 Equation (5)]) shows that TZp is weakly coercive. Therefore the sets 
^n{P,y,F,t) are weakly compact for all t > (cf. Remark 13. 2p . Thus, Theo- 
rem O shows that I](/3, y, F) =^ 0. 

Taking into account Example 14.51 it follows that TZp, S, and F satisfy As- 
sumption 3TJ Consequently, Items O and [3] follow from Proposition 14.31 □ 

In the following, we derive two types of convergence rates results with respect 
the £2_norm: the rate 0{^/S) (for p e (1,2)) and the rate 0{6^/p) (for every 
p € [1,2)) for sparse sequences — here and in the following, x'^ G ^^(A) is called 
sparse, if 

supp(a;t) := {A e A : 4 7^ 0} 

is finite. The same type of results has also been obtained for sparse Tikhonov 
regularization in [23l [42] . 
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Proposition 6.4. Letl <p <2, = {x{)xeA G ^^(A), and letF: P{K) ~* Y 
he a bounded linear operator. Moreover, assume that there exists uj Cz Y with 
97^p(a;1■) = F*uj. 

Then there exists a constant dp > only depending on p such that 
snp{\\xp - a;t||£2(A) : xp G T,{P,y,F)} 

for all j3 > and y eY with \\F{x^) - y||y < p. 

Proof. The assumption dTZp{x'') = F*u! implies that 

-{dTZp{x''),x-x'') < \{dTZp{x''),x-x^)\ < \\oj\\y\\F{x - x^)\\y ■ (44) 

Thus (|3T|) holds with 71 = and 72 — \\i^\\y- Theorem 15 . 5 1 therefore implies the 
inequality 

sup{Z?a7J,(.t)(x/3,xt) e S(/3,2/,F)} < My{P + \\Fx^ - yWv) . (45) 
From |231 Lemma 10] we obtain the inequality 

""""'"'^(^) - 3T^^;(^yT^''^^''(^^)("'"') 

for all a; € dom(7^p). Now, (gS]) follows from ^ and dH]). □ 

Remark 6.5. Since ^^(A) is 2-convex (see [31]) and continuously embedded in 
£^(A), Proposition 15.81 provides an alternative estimate for xp — a;^ in terms 
of the stronger distance ||-||fp(A)- The prefactor in (|39p . however, is constant, 
whereas the prefactor in (|43)l tends to as Tlp{x'') increases. Thus the two 
estimates are somehow independent from each other. ■ 

Proposition 6.6 (Sparse Case, p > 1). Let p e (1,2), let x^ — {x\)xeA G 
€^(A) be sparse, and let F: €^(A) Y be bounded linear. Moreover, assume 
that there exists ui Q Y with &R.p{x'^) = F*Ld and that F is infective on 

V := {x e e'^{A) : supp(x) C supp(a;^)} . 

Then 

sup{\\xp ~ x+|U2(A) : xp G j:{/3,y, F), \\Fx^ - y\\Y < /?} - OiP^/") as 13 ^ . 

(47) 

Proof. The injectivity of F on the finite dimensional space V implies the exis- 
tence of a constant C > such that 

C\\F{x)\\y > ||a;|l«2(A) , for all x £ V . 
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Denote hy tt, tt-^ : X ^ X the projections 

TTX := ^ x\ex, TT^x ^ xxex. 

AGsupp(a:''' ) A^supp{a^''' ) 

By means of the inequality (a + b)P < 2P-^{aP + bP) for a, & > 0, it follows that 
for every x £ £'^{A) 

\\x - xt||^,,. , < 2P-' \\nix - xt)r,,,, + 2P-mn^x\\% 



< 2P-'CP\\F{nix - xt))||P + . 
Applying the inequality (a + 6)^ < 2P^^(a^' + bP) a second time shows that 



||F(^(x-xt))||P < 2P-i||^^(x-xt)||^ + ||^^^^(x-a;t)|jP 

<2P-^F{x-x^)\\Py + \\F\\P\\n^x\\P.^^y (49) 



In [23] it is shown that 



ll7r-^a;||^2(A) < 7^p(7^-La;) < L»97^(^t)(a;, x^^) . 
Together with inequalities (|48|) and (|49p . this implies 

- a;^||^2(A) 

< 2^(P-^^CP\\F{x - x^)\fy + 2P^^(l + 2P-^CP\\F\\P'^ h^x\fp 

< 2'<^P-'^CP\\F{x - x^)\\^^ + 2P-'{l + 2P-'CP\\F\\p)Dsni.^^{x, x^) 



£2(A) 



(50) 



As in the proof of Proposition 16.41 one verifies that the inequality holds for 
ally eY with \\F{x'<) - y\\ < p. Therefore gT]) follows □ 

Next we derive the rate 0{S) for p = 1. 

Proposition 6.7 (Sparse case, p — 1). Let — (x|)a6A G i'^{A) be sparse. 
Assume that there exist ^ = (^A)AeA G dTZi{x'^) and ui £ Y with ^ = F*u!, and 
that F is injective on 

V^{xE e{K) : supp(x) C {A e A : |a| = 1}} ■ 

Then 

sup{|jx0-a;t||p,^) :xp(^^{[3,y,F), \\Fx^ - yWy < P] = Oifl) as /3 ^ . 

(51) 



Proof. Analogously to the proof of ([50]) one shows that 

\\x - x^Wp^k) < C\\F{x - x^)\\y + (l + Cll^^ll) ||7r^a;||£2(A) (52) 
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holds for every x E £'^(A). Further, in [53] it is shown that 



k^a;||£2(A) 



D 



< 



l-max{|eA| :Ae A,|a| <1} ' 
Finally one verifies as in the proof of Proposition 16.41 that 

^aK,(.t)(x^,xt) < My{P+\\F{x^)~v\\y) 

holds for every xp S 'S,{/3,y, F). Combining the estimates ([& 
shows ([5T|) and concludes the proof. 



(53) 

(54) 
and IE 



□ 



Remark 6.8. Let p e [1,2). If V is any finite dimensional subspace of £^(A) 
and F is injective on V, then there exists a constant Cp > such that 



Cp\\F{x)\\y > \\x\\iP(A-) , 



for all X €z V . 



Arguing as in the proofs of Propositions 16.61 and 16.71 one therefore derives the 
convergence rate 

sup{ \\xfs - xt ||,,(^) : xp e I](/3, y, F), \\Fx^ - y||y < /?} = 0(/?) as /3 ^ 
for the reconstruction of sparse sequences . ■ 

The convergence rates results for constrained £p regularization are summa- 
rized in Table [T] 



Rate 


Norm 


Premises (besides ran(i^*) n dlZp 7^ 0) 


Result 






(1,2) 


Prop. EH 






(1,2) 


Rem. 16.51 






p e [1,2), sparsity, injectivity on V 


Props. EH EJl 




V{ 


p € [1,2), sparsity, injectivity on V 


Rem. EH 


P 


Up 


p S (0, 1), uniqueness of x\ 
sparsity, injectivity on V 


Prop. |6.11| 



Table 1: Convergence rates for constrained £''' regularization. 



6.2 Non-convex Regularization 

We now drop the requirement that p > 1 and instead choose < p < 1. That is, 
we consider the minimization problem ((42|l with < p < 1 . In the following we 
show that most results of the superlinear case p > 1 carry over to the sublinear 
case. Note, however, that the functional TZp is non-convex for p < 1 and thus 
its restriction to a set of the form {x £ £p : \\Fx ~ y\\ < /?} may have local 
minima. Also, the 7?.p-minimizing solution of Fx = y need not be unique. 



23 



Proposition 6.9 (Well-Posedness). Let F: P{K) Y he a hounded linear 
operator with dense range. Then constrained P' regularization with < p < 1 is 
well-posed: 

1. For every /? > and y (z Y the set of solutions y, F) is non-empty. 

2. Let —* P > and yk y ^ Y. Then ill ^ Tp-Limsup^ ^{l3k,yk, F) C 

3. Let \\yk — y\\ < Pk ^ and assume that the equation Fx = y has 
a solution in i^^A). Then % ^ Tp-Limsup^ y^, i^) C E(0,y,F). 
// the equation Fx = y has a unique TZp -minimizing solution , then 
Tp-Lim^ E{f3k,yk, F) = {x^. 



Proof. This is similar as Proposition 16.21 □ 

Remark 6.10. In case the 7?.p-minimizing sohition a;^ of Fx — y is unique, 
Item [3] in Proposition 16.91 imphes that 

sup{||a;/3 -a;t||£p(A) : Xfi e y p , F) , Wy-ypW < /3} -> as /3 ^ ; 

otherwise there would exist a sequence /3fc ^ 0, S F with ||?/fc — ?/|| < Pk 
and e > such that \\xk — x'^\\£p(a) > £ for some Xk € Y,{Pk,yk, F), k G 
N. This, however, contradicts the assertion of Proposition 16.91 that {x^} = 
Tp-Um^,T.{Pk,yk,F). m 

Now we prove convergence rates for non-convex £p regularization. Similar, 
but weaker results have been derived in [551 EI] in the context of Tikhonov reg- 
ularization. In [51] , the conditions for the convergence rates result are basically 
the same as in Proposition 16. Ill but only a rate of order 0{y/S) has been ob- 
tained. In [22], a linear convergence rate 0{5) is proven, but with a considerably 
stronger range condition: each standard basis vector ca, A e A, has to satisfy 
e\ e rani^*. 

Proposition 6.11. Let F: £^(A) ~> Y be a hounded linear operator with dense 
range and let x^ = {x^^)\^\ S ^^(A) he sparse. Assume moreover that there 
exists a unique TZp-minimizing solution x^ of Fx — Fx^ , and F is infective on 

V := {x e e'^{A) : supp(x) C supp(a;^')} . 

Then 

sup{||x0-a;^|l,2(A) ■.xpeE{P,y,F), \\Fx^ ~ yWy < P} ^ 0{p) as P ^ . 

Proof. Denote supp(a;t), which by assumption is a finite set, and let tt, 

TT^: ^^(A) f{K) be the projections 

TTx = ^ xxex , TT^x = ^ xxex ■ 

Aen A^si 
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As in the proof of Proposition [ST7| (see ([5^ ). we obtain the existence of Ci > 
such that 

- a;t||,.(A) < Ci||F(a; - + (l + Ci ||^^||) ||7r^a;||,2(A) (55) 

for every x € P{A). 

We now derive a bound for ||7r^x||f2(7\^') . For every a E {±1}^, we define 
C(a) G em by 

C(a)A i-aAixlrS xen. 

Note that C(o')a is well-defined, because by assumption x\ for every A G 57. 
Now let a; G £'^{A) and let (T\ = sgn(a;A — x\) for every A G with x\ ^ x\ and 
a\ = 1 ii x\ — x\. Then 

7^,(xt)-7^p(7^x) = 5](|xt|^'-|x.r) < J2 (i4r-KP 
< E i4r'i4-^Ai<Ei4r'i4--^Ai = (c(^),^(.x-xt)). 

a;t|>|a;A| ^eQ 

Consequently, 

max |(C(CT),7r(a; - x''))\ > Tlp{x'') - 7^p(7ra;) for every x G ^^(A) . (56) 

cre{±l}'"' 

Denote by i: £^(f2) i'^(A) the embedding i{x) = x. Since Foi is injective 
and «* = TT, it follows that (Foi)* = Troi^* is surjective (see (501 Cor. VII. 5. 2]). 
In particular, C((t) G ran(7r o F*) for every a G {±1}^. Thus, there exists for 
every a G {±1}^^ some uj{a) G Y such that tt o F*a;(cr) = C(o')- With the 
abbreviation 

C2 := max ||w(CT)|ly, 
o-e{±i}" 

we therefore obtain that 

I (C(o'), 7r(a; — x^)) I | (vr o _F*aj(cr), 7r(a; — a;^)) | 

< \{F*uj{<t),x- x^)\ + \{F*uj{cr),7r^{x-x''))\ 

< I {cj{a), F{x - x^)) I + ||F*c.(a)||p(A) ||7r^(a; - xt)|j,2(A) 
<C2|jF(a;-a;t)||^ + C^2||F|||K^x||,2(A) 

for every x G £^(A) and a G {±1}^. Together with (|56p . this implies that 
C2||F(a;-xt)||y > max |(C(a), 7r(a; - xt)) | - C2||F|| ||7r^x||,2(A) 

<tG{±1}" 

> 7^p(a;t) - TZpinx) - C2||F|| ||7r^x||,2(A) (57) 

for every x G £^(A). 
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Since p < 1, there exists e > such that 

(C2IIFII + l)|t| < \t\P whenever t e R with |t| < e . 
Thus, for every x £ £^{A) with ||7r^x||£oo(A) < e we have 

{C2\\F\\ + < (C2IIFII + l)||7r^a;|l,i(A) 

= Y.{C2\\F\\ + l)\x,\ < Y,\x,\P = Tl,{n^x) . 

Thus ([57]) imphcs that, for every x £ £^(A) with ||7r-'-a;||^=o(A) < £, 

7^p(xt) - n.p{x) + ||7r^a;||,2(A) < C2||i^(x - x^)\\y (58) 

Remark l6.10l and the assumption that is the unique 7?,p-minimizing solu- 
tion of Fx = Fx'^ imply that 

sup{||x^-a;t||,2(A) : a;;3 e - </?}-> as /3 ^ . 

Since H^^^g — a;^||foo(A) ^ ll^^^g — xt||^2(A), it follows that there exists /3o such that 
\\xfi - x^Wi^iK) < e for all xp € \\Fx^ ~ vWy < and (3 < (3^. 

In these cases, (|58l) applies. The estimates (l55t and ((58)) . and the inequalities 
T^p{xp) < 7?.p(a;'l') and ||-F'(x/3 — < 2/3 therefore imply that 

\\x^ - a;t||,2(A) < - xt)||y + (1 + Ci||F||)||7r^x^|l,2(A) 

<Ci||^^(:r^-xt)||y 

+ (1 + Ci\\F\\)C2{\\F{xp - + -Rpixp) - 7^p(xt)) 
< (Ci+C2+CiC2||F||)||F(a:^-xt)||y 
<2{Ci+C2 + CiC2\\F\\)l3, 

which proves the assertion. □ 

7 Conclusion 

Due to modeling, computing, and measurement errors, the solution of an ill- 
posed equation F{x) = y, even if it exists, typically yields inacceptable re- 
sults. The residual method replaces the exact solution by the set S(/3,y,F) = 
argmin{7?,(a;) : S{F{x),y) < /3}, where 7?. is a stabilizing functional and iS de- 
notes a distance measure between F{x) and y. This paper shows that in a very 
general setting E{f3,y,F) is stable with respect to perturbations of the data y 
and the operator F (Theorem 13.61 and Proposition 13.10( 1. and the regularized 
solutions converge to 7?.-minimizing solutions of F{x) — y as /3 (Proposi- 
tion 13. 9p . In particular the stability issue has hardly been considered so far in 
the literature. 
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In the case where F acts between linear spaces X and Y , stabihty and con- 
vergence have been shown under a Hst of reasonable properties (see Assump- 
tion 14. ip . These assumptions are satisfied for bounded linear operators, but 
also for a certain class of nonlinear operators (Example 14. 6p . If y is reflexive, 
X satisfies the Radon-Riesz property, F is a linear closed operator, and TZ and 
S are given by the norms on X and Y, the set S(/3,?/,F) consists of a single 
element Xfj. This element is shown to converge strongly to the minimal norm 
solution as /3 ^ 0. In this special situation, norm convergence has also been 
shown in Theorem 3.4.1]. 

In Section [5] we have derived quantitative estimates (convergence rates) for 
the difference between x'^ and minimizers Xfj G Ti{(3,y,F) in terms of a (gener- 
alized) Bregman distance. All these estimates hold provided 5(F(x^),j/) < (3 
and a source inequality introduced in [26] is satisfied. For linear operators, the 
required source inequality follows from a source wise representation of a subgra- 
dient of 7?. at . This carries on the result of [4] for constrained regularization. 
In the case that X is an r-convex Banach space with r > 2 and TZ is the r- 
th power of the norm on X, we have obtained convergence rates 0{(3^/^) with 
respect to the norm. The spaces X — L'p{VL) for p £ (1,2] are examples of 
2-convex Banach spaces, leading to the rate 0(^/5) in those spaces. 

As an application for our rather general results we have investigated sparse 

regularization with p G (0,2). We have shown well-posedness in both the 
convex {p > 1) and the non-convex case {p < 1). In addition, we have studied 
the reconstruction of sparse sequence. There we have derived the improved 
convergence rates 0(/3^/p) for the convex and C(/3) for the non-convex case. 
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